Successive phase transitions and phase diagrams of the quasi-two-dimensional 
triangular antiferromagnet Rb4Mn(Mo04) 3 
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Comprehensive experimental studies by magnetic, thermal and neutron measurements have clar- 
ified that Rb4Mn(Mo04)3 is a model system of a quasi-2D triangular Heisenberg antiferromagnet 
with an easy-axis anisotropy, exhibiting successive transitions across an intermediate collinear phase. 
As a rare case for geometrically frustrated magnetism, quantitative agreement between experiment 
and theory is found for complete, anisotropic phase diagrams as well as magnetic properties. 

PACS numbers: 75.10.Hk, 75.40.Cx, 75.50.Ee 



Geometrically frustrated magnetism has been a subject 
of active research in condensed matter physics. Gener- 
ally in physics, quantitative comparison between exper- 
iment and theory is crucial to make a firm progress of 
our understanding. In the field of frustrated magnetism, 
however, such fortuitous cases are still scarce where full 
or semi quantitative agreement between experiment and 
theory have been found, except a few examples such as 
spin ice [1] and the orthogonal dimer SrCu2(BOs)2 Q- 

Two-dimensional (2D) triangular antiferromagnets 
(TAFM) have been extensively studied, because of rich 
frustrated magnetism expected on the simple 2D Bravais 
lattice Theoretically, it has gained a consensus 

that the ground state for the nearest-neighbor antiferro- 
magnetic (AF) Heisenberg model has the 120° spin order 
In this case, the concept of vector chirality, the 
handedness of the way the spins are rotated in a 120° or- 
der for a given triangle, may become essential and lead to 
exotic phenomena such as phase transitions with a new 
universality class [10] and multiferroic phenomena [Tlj . 

In fact, the vector chirality is expected to play an im- 
portant role in phase transitions on a triangular lattice, 
e.g., for the nearest neighbor Heisenberg model: 

H = J^S i .S j -D^{S;) 2 -g» B y 52H.S i , (1) 

i i 

where J > and D are intralayer exchange interaction, 
and single-ion anisotropy, respectively. Depending on the 
anisotropy, namely, the sign of D, the model exhibits 
three types of phase transitions, featuring effects due to 
the vector chirality. For Heisenberg spin with D = 0, 
the chirality forms a point defect called Z2 vortex, and a 
proposal has been made on its binding-unbinding transi- 
tion at finite temperature [12[. For XY type (D < 0), a 
long-range order of the vector chirality without dipole 
magnetic order is expected at slightly higher temper- 
ature than the Kosterlitz-Thouless transition into 120° 



quasi- long-range order [13[. For easy axis case (D > 0), 
successive transitions are expected associated with re- 
spective orde ring of the longitudinal and transverse spin 
components jl4 fl^. Namely on cooling, the system 
first forms a collinear intermediate phase (IMP) with the 
three-sublattice 'mid' structure [16[, and then transits 
into a 120° spin-order phase with a uniform vector chi- 
rality. Recent theoretical study has found another phase 
transition in the IMP, which separates the lower T 'mid' 
phase and a higher T collinear phase with three different 
sublattice moments. However, the latter phase is only 
stable in the purely 2D limit, and thus with finite inter- 
layer coupling, the 'mid' phase should become dominant 
throughout the collinear IMP [l^ . 

For this easy-axis case, unlike the Heisenberg and XY, 
experiments have confirmed the theoretical predictions. 
Namely, the successive transitions and/or 1/3 magneti- 
zation plateau have been observed for TAFMs with easy- 
axis anisotropy such as VCI2 [17], ACr02 (A = Li, Cu) 
[H, 0, and a metallic TAFM GdPd 2 Al 3 Q. How- 
ever, neither detailed study of the phase diagram under 
external field nor quantitative comparison between ex- 
periment and theory has been made so far, because of a 
relatively large scale of J and/or lattice deformation due 
to magnetostriction. 

Here, we report a comprehensive study on the crys- 
tal/spin structures and thermodynamic properties of the 
quasi-2D Heisenberg TAFM Rb 4 Mn(Mo0 4 ) 3 (RMMO). 
This material exhibits the successive transitions and 1/3 
magnetization plateau phase under field, reflecting its 
easy-axis anisotropy. The relatively small exchange cou- 
pling J allows us to construct complete phase diagrams 
for the first time under field both parallel and perpen- 
dicular to the easy axis. As a rare case in geometrically 
frustrated magnets, quantitative agreement between ex- 
periment and theory has been found for the phase dia- 
grams and magnetic properties, establishing the system 
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FIG. 1: (color online) Crystal structures of Rb4Mn(Mo04)3 featuring (a) MnOs polyhedra, (b) equilateral triangular lattices of 
Mn 2+ and M0O4 tetrahedra. Intralayer and interlayer distances between Mn 2+ ions are given by a — 6.099 A and c/2 = 11.856 
A respectively, using the lattice constants at 298 K. Phase diagrams of Rb4Mn(Mo04)3 for (c) jjloH || c and (d) jjloH \\ ab 
constructed by using various experimental techniques, and by Monte Carlo simulations (MCs) for D/J = 0.22 (broken lines). 



as a model 2D Heisenberg TAFM characterized by the 
Heisenberg Hamiltonian of Eq. (1). 

Single crystals were synthesized by a flux method [2l[ . 
The structure was determined by single crystal X-ray 
diffraction and adopts P63 / rarac symmetry (Ri = 2.88 
%). Powder neutron diffraction (PND) measurements 
were performed on the BT1 at NIST, and confirmed that 
the structure is consistent with the X-ray results and sta- 
ble down to 1.5 K. Thus, RMMO has the equilateral tri- 
angular lattice formed by Mn 2+ . Each Mn 2+ ion locates 
in a MnOs polyhedron and has a high spin ^g e g state, 
providing a S = 5/2 Heisenberg spin. The dominant in- 
tralayer coupling J should be made by the superexchange 
path Mn-O-O-Mn involving two oxygen atoms. The in- 
terlayer interaction should be negligibly weak because of 
the large separation between Mn 2+ ion layers due to two 
Rb + ions and two M0O4 tetrahedra. 

D.c. M was measured by a commercial SQUID mag- 
netometer above 1.8 K, and by a Faraday method for 
0.37 K < T < 2 K [22I] . Specific heat Cp was measured 
by a thermal relaxation method down to 0.4 K under 
fields up to 9 T. Pulsed field measurements of M were 
performed up to 27 T. Classical Monte Carlo simulations 
(MCs) were made using the standard heat-bath method. 

As a guide to understand the results, we first present 
in Figs. [He) and (d) T-H phase diagrams consisting of 
six phases (A)-(F). Each symbol represents a transition 
point determined by different probes. In the following, 
we describe details of the experiments used to construct 
the phase diagrams, and compare them with theory. 

We start with the specific heat measurement results. 
Notably, a double kink structure is found in the temper- 
ature dependence of the zero field Cp (Fig. [2]). Further- 
more, by applying field fioH || c, the two kinks become 
separated to form two peaks, providing clear evidence 
for the successive transitions. Here, we define Tni and 
Tn2 by the locations of the lower and higher tempera- 



ture kinks or peaks, respectively. Under field hqH || c, 
the IMP in between Tni and Tn2 becomes stabilized, 
while under ji^H \\ ab, Tni approaches Tn2 with increas- 
ing field, and finally the IMP disappears under n^H > 4 
T (Fig. [2Kb)). A broad tail of the peak seen at T > T N2 
suggests an enhanced 2D spin fluctuations. 

The successive phase transitions are also observed in 
the susceptibility x = M/H (Fig. [3]), particularly as two 
peaks of the temperature dependence of dx/dT under low 
fields (inset of Fig. 09(a)). On the other hand, only a sin- 
gle peak is found in the high field regions of Phase (B) for 
[IqH II c and (E) for fioH \\ ab. These peak temperatures 
are indicated by arrows in Fig. [3] and are found to overlap 
the phase boundary given by the specific heat anomalies, 
as shown in Figs. [H(c) and (d). The temperature depen- 
dence of both x an d Cp/T, and the slope of the phase 
boundary are found consistent with Ehrenfest's relation 
defined by A((dM/dT) H /dT) = -dT N /dHA(C P /T). 
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FIG. 2: (color online) T dependence of Cp/T under various 
fields for (a) fioH \\ c and (b) /joH || ab. Values of Cp/T under 
fields are shifted upwards for clarity, (c) Tni (black arrow) 
and Tn2 (red arrow) as a function of D/J obtained by MCs 
for J = 1.2 K. The horizontal lines indicate Tni = 2.42(2) K 
and Tn2 = 2.80(2), obtained from Cp/T at /jqH = 0. 
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FIG. 3: (color online) (a) c-axis and (b) a6-plane components 
of the susceptibility x(T) measured under various fields using 
zero-field-cooled sequence. Red and black arrows indicate Tni 
and Tn2 obtained by peaks in dx/dT. Insets: (a) dx/dT vs. 
T under fioH || c, (b) anisotropic x(T) obtained by MCs for 
typical fields used in experiment shown in the main panels. 



The high temperature susceptibility shows the Currie- 
Weiss behavior. The fitting using the formula C/(T + 
8w) in the range of 50-350 K yields the effective mo- 
ment p e ff = 5.95 (5.97) fiB for hqH \\ ab(c), close to the 
expected value 5.92 ji B for S = 5/2, and AF Weiss tem- 
perature 9 W = -20.3(5) (-19.6(5)) K for fi H \\ ab(c). 
The average 6w ~ —20 K corresponds to J = 1.14(5) 
K. The frustration parameter |0w|/^N2 = 7.14 is large 
and indicates geometrical frustration. The anisotropic 
ratio Xc/Xa6, which is constant and near unity at high 
temperatures, becomes larger than unity below ~ 10 K, 
suggesting that RMMO has the easy-axis anisotropy. 

Clear evidence for the easy-axis anisotropy is found in 
the field dependence of M. Figures [H presents the M(H) 
curve measured at 1.3 K as well as the results obtained 
by MCs. At T < Tn2, a magnetic plateau was observed 
under (iqH \\ c at ~ 1/3 of the saturation moment of 
5 /ib for Mn 2 + (Fig. [1(a)). Meanwhile, for fi H || ab, 
M increases monotonically with field and fully saturates 
to ~ 5/iB (Fig. Hfb)). Quantitative agreement with the 
MCs results is found except a slight deviation around 
the plateau region for ji^H \\ c. Because T ~ J, the fi- 
nite T effect broadens the anomalies at the critical fields. 
Therefore, by using the kink found in dM/dH, we de- 
fine the lower and upper critical fields of the plateau re- 
gion, /jLqH c i and /io# C 2> and the critical field associated 
with the saturation of magnetic moment, /ioH c s. The 
1/3 plateau of the c-axis M(H) as well as its larger slope 
than the afr-plane M(H) clearly indicates the easy- axis 
anisotropy. As can be seen in the phase diagram of Fig. 
[2(c), the plateau field region found by the M(H) curve 
becomes systematically wider with increasing tempera- 
ture, and is smoothly connected with the IMP found by 
the temperature dependence of both Cp/T and x- This 
provides experimental evidence that Phase (B) has the 
'mid' structure (Fig. [2(c)). 



FIG. 4: (color online) Field dependence of the magnetization 
and its derivative for Rb4Mn(Mo04)3 at T = 1.3 K measured 
under a pulsed field (solid lines) and calculated using MCs 
(broken lines) for (a) fioH \\ c and (b) /joH || ab. 

By comparing experiment with theory, we first esti- 
mate the parameters J and D. The mean- field theory 
predicts that the afr-plane and c-axis M are linear in H 
and follow M = N A (gfi B ) 2 SH/(9J - 2D) up to fi H c3 
and M = N A (gfi B ) 2 SH/(9J - 6D) up to fJL H cl , respec- 
tively [Hj]. The experimental results in Figs. [U(a) and 
(b) are indeed i^-linear and the fits give J = 1.2 K, D 
= 0.28 K and thus D/J = 0.23. J has nearly the same 
value as the one obtained from 0w- Then fixing J = 1.2 
K, and comparing zero field Tni and Tn2 with those ob- 
tained by MCs as a function of D / J, we estimate D / J to 
be 0.22(2), as indicated by a vertical broken line of Fig. 
[2](c), consistent with the above estimate. Thus through- 
out the paper, we adopt J = 1.2 K and D/J = 0.22. 

Now, we compare the phase diagrams obtained by ex- 
periment and by the MCs performed for the 2D Heisen- 
berg TAFM model represented by Eq. (1) (Figs. [2(c) 
and (d)). The agreement between experiment (sym- 
bols) and theory (broken lines) is significantly good in 
detail for both field directions including the magnetic 
plateau phase. This suggests that the spin Hamiltonian 
for RMMO is well captured by Eq. (1), and the possible 
extra terms, for example, Dzyaloshinsky-Moriya interac- 
tion, should be negligibly small in comparison with the 
J and D terms. Three-sublattice spin structures inferred 
from both experiment and theory are schematically pre- 
sented by solid arrows in each region of Phases (A)-(F). 

At the ground state, the theory predicts a 120° struc- 
ture with slight canting toward the c-axis due to the easy- 
axis anisotropy, as schematically shown in Phase (A) of 
Fig. [2(c). The canting angle is estimated 2.7° for H = 
by the relation cos(tt/3 -9) = 3J/(6J - 2D) [15]. This 
canting causes an increase in x(^) below Tni, as con- 
sistently seen in both experiment and theory (Figs. [3](a) 
and [51(b) inset). According to theory, this canted struc- 
ture is stable under ji^H \\ c in Phase (A). With further 
increasing ji^H \\ c, however, the 'uud' structure with 
a 1/3 magnetization plateau takes over in Phase (B) as 
observed in experiment, and then should transit into the 
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FIG. 5: (color online). Difference between the powder neutron 
diffraction obtained at 1.5 K and 10 K (red) together with the 
calculation for the quasi-2D 120° spin structure (black). 



agreement between experiment and theory for geometri- 
cally frustrated magnetism. Our material should serve 
as a model system to further study novel 2D frustrated 
magnetism, such as critical dynamics associated with vec- 
tor chirality, multiferroic noncolliner magnetism, and the 
possible magnon decay in the dispersion spectrum j23[. 
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'oblique' phase (C) with two parallel spins and one point- 
ing to a different direction. Finally in Phase (D), the 
moments become fully polarized. 

Because D/J <C 1, the ground state canted structure 
at H = should be nearly degenerate with a fan-shape 
structure having one spin in the afr-plane [l5[. This 
fan-shape structure has a weak ferromagnetic compo- 
nent in the afr-plane, and can be easily stabilized un- 
der a weak in-plane field in Phase (E), inducing an en- 
hancement of x(T) below Tni. For this behavior as well, 
we find a quantitative agreement between experiment 
and theory (Fig. [3f b) and its inset). Notably, the fan- 
structure in Phase (E) a uniform vector chirality pointing 
to an in-plane direction perpendicular to the field. Under 
H$H II a6, the IMP(F) becomes narrower in temperature 
where the theory predicts that a collinear spin structure 
becomes inclined toward the field direction (Fig. [2(d)). 

In order to confirm the 120° structure under zero field, 
we performed the PND measurements on the BT7 at 
NIST. Figure [5] shows the difference between the PND 
spectra obtained at 1.5 K and 10 K. A magnetic peak 
at the wave vector ~ (1/3, 1/3, 1) indicates the 120° in- 
plane spin structure as well as AF inter layer correlations. 
We fit the data to the analytical formula for the spheri- 
cal average of magnetic scattering from a quasi-2D 120° 
magnetic structure with tiny canting component corre- 
sponding to Phase (A). The out-of-plane correlations are 
described by Lorentzian and the in-plane correlations 
by Lorentzian squared [6]. Consequently, the in-plane 
and out-of-plane correlation lengths are estimated to be 
£ab > 98(1) A « 16a and £ c « 16.3(5) A « 0.7c, re- 
spectively, indicating quasi-2D magnetism and weak in- 
ter layer correlations. While further neutron experiments 
using a single crystal are necessary to more precisely de- 
termine the correlation function forms and lengths, the 
present observation provides a basis for the comparison 
between experiment and theory. 

Our study has established a rare case of quantitative 
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